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METHOD AND SYSTEM FOR SIMULATION OF SURGICAL PROCEDURES 

Field of the Invention 

The present invention generally relates to simula- 
tion of surgical procedures and, in particular, to simu- 
5 lation of minimal invasive surgical procedures in a vir- 
tual environment . 

Background Art 

In modern surgery minimal invasive techniques are 

10 used in more and more applications. The development of 
technology within this relatively new field advances 
quickly, which results in great training requirements for 
surgeons. One way of rendering the training more effec- 
tive is to use computer simulations. Known techniques of 

15 providing a credible simulation are very complicated and 
expensive with respect to computer utility in the form of 
processor and memory. Moreover, the result is not suffi- 
cient to provide a realistic simulation environment. The 
visual properties that the anatomy exhibits in reality 

20 are difficult and time-consuming to recreate in a simula- 
tion. 

Within the field of laparoscopy a camera is used to 
supply picture information from the patient to the sur- 
geon. The display screen shows the picture that the cam- 

25 era catches of the inside of, for example, the abdominal 
cavity. All the instruments and the anatomy with which 
the surgeon works are reproduced by means of the camera 
and the display screen. The surgeon uses the information 
on the display screen to control and operate his or her 

30 instruments and carry out the procedures which are re- 
quired to perform the surgical operation. Since the mini- 
mal invasive techniques supply information to the surgeon 
by means of a display screen, the three-dimensional real- 
ity is reduced to two dimensions on the display screen. 
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The picture therefore lacks, among other things, the in- 
formation as to depth that exists in reality. The surgeon 
has to make up for this loss of information by studying 
lighting conditions, colours, etc. 
5 By means of modern computer engineering, it is pos- 

sible to provide realistic training situations in a vir- 
tual environment created by a computer program. In the 
computer program, a three-dimensional model of the object 
which the simulation concerns is provided. The user is 

10 provided with a projection thereof which should corre- 
spond to the picture information which in a real situa- 
tion is caught by a camera. This means that all visual 
information, such as instrument and anatomy, is drawn by 
the computer. However, there are still considerable dif- 

15 ferences between the picture information that the com- 
puter tries to recreate and the actual picture that a 
camera catches. A large part of the picture information 
that the surgeon uses in real life, for example light ef- 
fects and anatomic structures, factors such as breathing 

20 and beat of the pulse, which are present in the real 

situation are difficult to recreate realistically in the 
computer . 

Many complicated processes within the field of sur- 
gery are expensive, if not impossible, to fully simulate 

25 in a computer. This means that the simulation can only 
consist of short procedures which lack the continuity 
that is usually present when the process is carried out. 
For instance, in US 5,791,907, Ramshaw et al disclose a 
system for simulation of an entire operation. This solu- 

3 0 tion is founded on a database of recorded video sequences 
which visualise different procedures in an operation. The 
actual simulation then consists of the user being intro- 
duced to a video sequence which shows a separate proce- 
dure. Subsequently, the user is asked a number of ques- 

35 tions. Then a video sequence is played showing the con- 
tinuation of the operation which is determined by the an- 
swers to the questions. 
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Object of the Invention 

The object of the present invention is to provide a 
method and a system which create a realistic simulation 
5 environment, and which wholly or partly solve the above- 
mentioned problems. A further object is to provide a 
method and a system for a realistic simulation of a com- 
plete operation. 

This is achieved by a method and a system according 
10 to the appended claims. 

Summary of the Invention 

The present invention relates to a method for simu- 
lating a surgical procedure on a bodily object, such as 

15 an organ, in a virtual environment provided by a computer 
unit. The environment comprises a three-dimensional 
mathematically described model of said bodily object, 
which reflects at least the geometrical and mechanical 
properties of said bodily object, and a virtual instru- 

20 ment which is controlled by a physical feeding means con- 
nected to said processing unit, which makes it possible 
to affect said model. The method according to the inven- 
tion comprises the step of representing a two-dimensional 
projection of said model by means of a video sequence 

25 containing a recorded view of a real bodily object, en- 
gagement with the virtual instrument only interacting 
with said model . 

In a virtual environment, an object can be viewed 
from different angles, but limits are often introduced in 

30 order to resemble the real environment. For example, in 
an operation in the abdomen, limits can be introduced so 
that a user in the virtual environment can work only from 
the directions which. are physically possible in a real 
situation. In minimal invasive operations a laparoscopic 

35 camera is used to film inside the body. This results in 

the user limiting himself or herself to one or more view- 
ing angles. Consequently, the virtual environment, i.e. a 
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viewing angle, is determined and by means of this angle a 
two-dimensional projection is created from the three- 
dimensional model. This two-dimensional projection corre- 
sponds to that shown when a laparoscopic camera is film- 
5 ing inside a body. A bodily object here means something 
in the body that is subjected to the surgical procedure, 
for example, an organ such as the liver, the heart etc. 
or, for instance, a cancerous tumour. The computer unit 
can consist of a computer with associated equipment, for 

10 example a personal computer. 

The virtual instruments used are controlled by a 
physical feeding means. The feeding means may consist of, 
for example, instruments which resemble those used in a 
real operation, may be mounted in a stand fitted with 

15 sensors. One example of such a physical feeding means is 
a "Laparoscopic Impulse Engine" which is made by Immer- 
sion Corporation, USA. This allows five degrees of free- 
dom, i.e. rotation about a pivoting point (the point 
where the instrument is inserted into the body) , the pos- 

20 sibil ity of moving the instrument longitudinally inwards 
and outwards, rotation round the axis of the instrument 
and a gripping function. In order to allow the user to 
interact with the virtual object and show where the in- 
strument is positioned relative to the same, also the 

25 virtual instruments are shown on the display screen. Con- 
veniently, the initial position of the virtual instru- 
ments on the display screen is determined by the process- 
ing unit; subsequently, they are moved in the virtual en- 
vironment based on the movements with the feeding means. 

30 Since the viewing angle for the virtual object is 

fixed, the two-dimensional projection essentially becomes 
the same, notwithstanding movements due to pulse, breath- 
ing or intervention. If surface rendering is created with 
the purpose of obtaining a real appearance of the mod- 

35 elled object, it will be possible to simulate an opera- 
tion. As mentioned by way of introduction, this does not, 
however, result in a realistic environment. Thus, the 
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method according to the invention uses a recorded video 
sequence. The method according to the invention provides 
a realistic simulation environment which shows the natu- 
ral movements of an object, such as breathing and pulse. 
5 By replacing a computer-created projection of a three- 
dimensional model of a bodily object with a recorded 
video sequence, a considerably higher degree of realism 
is achieved. A fundamental idea behind the invention is 
that bodily organs in different human beings essentially 

10 have the same structure. However, the computer-based 

model of the organ essentially has to be similar to the 
organ in the film. Unlike other solutions, the invention 
is based on the similarities of organs, i.e. the inven- 
tion can be based on a generic model, for example, of a 

15 stomach and on this apply the video sequence. However, 

adaptations may be necessary either by the generic model 
per se being affected by the picture material or by the 
picture material being affected by knowledge of the prop- 
erties of the model. It is, however, important that the 

20 virtual instrument acts. on the computer model, and not on 
the video sequence. Since the video sequence contains a 
certain limited movement in the model such as breathing 
and beat of the pulse, the underlying model needs to be 
animated correspondingly so that an operation by means of 

25 a virtual instrument hits the same point in the virtual 
model as the point that can be seen on the video se- 
quence . 

In order to visualise changes in the computer model, 
changes in what is shown need to be made. One way of do- 

30 ing this is to provide a rendered surface of those and 
only those portions of said model which are affected by 
said virtual instrument, and to superimpose said rendered 
surface on said video sequence. Thus, a great advantage 
compared with known systems and methods is achieved since 

35 only those portions that have been changed need to be 

rendered. This saves computer utility but, above all, the 
realism is increased for the user as regards the visual 
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impression. Consequently, essentially the entire object 
is shown to the user as a filmed sequence, which is im- 
portant on the one hand as regards the user's possibility 
of recreating the three-dimensionality in the two- 
5 dimensional picture and, on the other, to increase the 
feeling of being "for real". Preferably, the video se- 
quence contains a whole breathing and pulse cycle. Since 
there are more beats of the pulse than breaths during a 
certain period of time, the video sequence suitably con- 

10 tains a sequence which starts when the beats of the pulse 
and the breathing take place at the same time and fin- 
ishes when the beats of the pulse and the breathing occur 
simultaneously. The reason for this is that the virtual 
environment should be as similar as possible to the real 

15 one. 

In addition, the invention allows what is shown of 
the virtual instruments to be affected by their interac- 
tion with the model. Since the video sequence is only a 
visualisation, while the virtual instruments interact 

20 with the model and, thus, uses the information as to 

depth in the model, information is obtained if the in- 
struments are wholly or partly hidden by protruding parts 
of the model. For example, a needle is partly hidden as 
it penetrates the wall of an organ or an instrument may 

2 5 be hidden by an organ that is located in the foreground 
of the object to be operated on. 

Since the virtual instrument works in a three- 
dimensional virtual environment, it is important to be 
able to analyse that the user correctly estimates how the 

30 three-dimensional environment is built based on the vis- 
ual impression of the two-dimensional projection. Pref- 
erably, the method therefore comprises the step of meas- 
uring at least one parameter in the virtual instrument's 
movements or interaction with the object, the parameter 

35 reflecting the user's skill. Parameters which can be 

measured to estimate the skill are time, number of colli- 
sions with tissues, number of incorrect procedures, etc. 



WO 02/059859 



PCT/SE02/00054 



7 

By following up the simulation in a structured manner, 
the possibilities of learning are improved, especially 
learning of procedures that the users have not observed 
that they made incorrectly at the moment of the simula- 
5 tion. 

Furthermore, the invention comprises a method for 
simulating a surgical operation containing a plurality of 
surgical procedures on a bodily object, such as an organ, 
in a virtual environment provided by a computer unit . The 

10 environment comprises a three-dimensional mathematically 
described model of said bodily object, which reflects at 
least the geometrical and mechanical properties of said 
bodily object, and a virtual instrument which is con- 
trolled by a physical feeding means connected to said 

15 processing unit, which makes it possible to affect said 
model. The method comprises the steps of representing a 
two-dimensional projection of said model by means of a 
video sequence containing a recorded view of a real bod- 
ily object, providing said model in a plurality of states 

20 which reflect the object during a number of distinct 

phases of an operation, providing a first plurality of 
video sequences which correspond to the natural movements 
of a real object during the respective states, providing 
a second plurality of video sequences which each visual- 

25 ise the surgical procedure that shows how the object is 
changed from one state to another, and of allowing the 
user to choose between simulating a procedure and looking 
at a visualisation of the procedure in order to change 
the object from one state to another. 

30 Consequently, the possibility is achieved of simu- 

lating a whole operation. The states which reflect the 
object during a number of distinct phases of an operation 
can, for example, consist of an initial state with an un- 
changed object, an object having an incision for suture 

35 or the object with a completed suture. By such a method a 
training process is provided, in which the continuity in 
a real procedure is saved by using real picture informa- 
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tion. This results in great advantages compared with sys- 
tems that are based on a solution purely involving video 
sections, such as the solution which is presented in 
US patent application 5,791,907 by the users themselves 
5 having the possibility in the process of choosing to make 
the procedures they desire to practice, whereas they can 
choose to just look at visualisations of the procedures 
that they already know well or do not handle. Besides, a 
larger degree of comprehensive understanding is achieved 

10 when separate surgical procedures are not simulated indi- 
vidually but put in their context. 

Furthermore, the invention comprises a system for 
simulating a surgical procedure on a bodily object, such 
as an organ, in a virtual environment provided by a com- 

15 puter unit. The environment comprises a three-dimensional 
mathematically described model of said bodily object, 
which reflects at least the geometrical and mechanical 
properties of said bodily object, and a virtual instru- 
ment which is controlled by a physical feeding means con- 

20 nected to said processing unit, which makes it possible 
to affect said model. The system comprises a two- 
dimensional projection of said model, which is repre- 
sented by a video sequence containing a recorded view of 
a real bodily object, engagement with the virtual instru- 

25 ment only interacting with said model. This system essen- 
tially exhibits the same advantages as the corresponding 
method mentioned above. 

Brief Description of the Drawings 
30 In the following, the invention will be described in 

more detail by way of example with reference to the ac- 
companying drawings, in which 

Fig. 1 shows a simulation environment, and 

Figs 2a -2b show a recording of a video sequence. 



35 
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Description of a Preferred Embodiment 

A simulation environment according to a preferred 
embodiment of the invention comprises a processing unit 
1, a display screen 2, a feeding means 3, which is shown 
5 in Fig. 1. The virtual environment is realised by means 
of the processing unit 1. The processing unit can, for 
example, consist of a personal computer which is prefera- 
bly equipped with a graphics card having 3D-circuits. 
Figs 2a-2b show how a real surgical procedure is 

10 filmed by means of a camera both before, during as well 
as after the procedure. In order to allow filming inside 
a body, a laparoscopic camera is used. Fig. 2a shows how 
a laparoscopic camera is filming inside an organ and the 
picture obtained is shown in Fig. 2b. Subsequently, this 

15 film will constitute the base ^of the visual properties in 
the simulation. Apart from the picture information, lumi- 
nous intensity, light angle and viewing angle are pref- 
erably recorded. From this film a section is selected, 
which constitutes a complete breathing and pulse cycle. 

20 Since it has the same beginning and end, it can be placed 
in a loop in order to continuously reflect the patient's 
natural movements. This means that there is a finite num- 
ber of filmed pictures which together constitute a filmed 
video sequence. This video sequence is to be used to show 

25 the natural movements of a simulation object which is not 
affected by the virtual instrument. In addition, the ac- 
tual operation is recorded, which makes it possible to 
visualise the whole operation. 

In one embodiment a computer model of the objects 

30 appearing in the film is then created, which can be an 
organ, cancer cells, etc., in the virtual environment, 
based on the visual impression and knowledge of the anat- 
omy of the object. This computer model becomes a three- 
dimensional model which is drawn relatively freely in a 

35 computer environment. In another embodiment a number of 

generic models of human organs and other objects have al- 
ready been created. Irrespective of the point of depar- 
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ture, it is on this model that the simulation will be 
based. It is thus important that the model has the geo- 
metrical, mechanical and tactile properties that a real 
object or organ has. Consequently, the model can be de- 
5 scribed entirely mathematically. The mathematically de- 
scribed data structure contains information about both 
surface and volume. The surface information can be used 
with the purpose of rendering and the volume information 
is necessary in order to learn about the mechanical and 

10 visual properties of the inside of the object. Examples 
of data structures are voxels or octtrees . 

It is important to obtain the same perspective of 
the modelled object as the filmed object. In the simplest 
embodiment, this is carried out by turning the model in a 

15 computer environment in order to optically determine when 
the correct perspective is found. In a more advanced em- 
bodiment there are algorithms which turn and/or scale the 
model until it corresponds to the filmed object. During 
the comparison, one or more pictures from the filmed se- 

20 quence are used. One way of comparing pictures from the 

film and projections of the model is to start with a pic- 
ture from the film and then proceed with more pictures 
from the film with the purpose of possibly compensating 
by turning the model somewhat . Another way of comparing a 

25 picture from the film with a specific perspective of the 
model is to compare sharp lines in the picture with the 
outlines of the model in the projection. In the same way, 
it is possible to determine if darker portions in the 
picture together with information about the location of 

3 0 the light source are positioned deeper down than another 
portion. These comparisons result in a given perspective 
being obtained, which is then locked to be determined as 
the same as that from which the real object is filmed. 

For the user of the simulation equipment, the lim- 

35 ited video sequence is now presented as a projection of 
the model. Thus, a completely realistic environment for 
practicing is achieved. Since the model is located behind 
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the film and the virtual instruments really operate in 
the model, full realism is achieved considering that the 
depth effect is fetched from the model and not the film. 
The film is presented to the user on the display 
5 screen 2 . 

The model and its visualisation through the video 
sequence now constitute the basis for the actual interac- 
tion with the user. In order to resemble a real situa- 
tion, a feeding means 3 is used, which essentially is 

10 identical to that used in a real operation. However, this 
is mounted in a stand and connected to sensors for de- 
tecting movements. As in a real operation, the user looks 
at the movements of the instruments on the same screen 2 
as that on which the visualisation of the model is shown. 

15 The instruments are represented as they would have looked 
like if they engaged with a real body. The difference is 
that the visualisation on the screen 2 does not originate 
from a laparoscopic camera, but from a visualisation cre- 
ated in the processing unit. The virtual instruments thus 

20 engage with the model, which allows feedback of power 
when the model contains the necessary data. Naturally, 
when real instruments engage with a real object, a change 
of the object occurs; for instance, an incision with a 
knife results in an incision in the object. Consequently, 

25 this is shown in a real operation on the screen which 
shows what the camera is filming. In the preferred em- 
bodiment rendered portions are created of the areas in 
the model which have changed visually in the perspective 
shown to the user. These are then superimposed on the 

30 video sequence. In order to make the rendered portions 

reflect the natural movements of the object, just as many 
rendered portions must be created as there are pictures 
in the video sequence. 

In order to create a realistic environment, there is 

35 also in the preferred embodiment means for affecting how 
the virtual instruments 3 are visualised on the screen 2. 
Since the virtual object is completely described in three 
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dimensions at each moment of the animated computer model, 
it is possible to determine when a collision between the 
virtual instrument and the virtual object takes place. In 
reality, a deformation of the object occurs which is 
5 highly evident, for example, when the tip of a needle is 
about to penetrate the tissue. The same effect is 
achieved by finding the point where the instrument hits 
the object and determining deformations. Subsequently, a 
new surface structure is determined in the deformed area 

10 and from information about the light source when filming 
the video sequence, a new picture can be created. This 
picture is placed on the film in a new layer and by using 
the transparency in the borders which can be obtained by 
means of special 3D-graphics cards, the superposition can 

15 take place with an extremely small visual border. By 

knowledge of the surfaces of the objects and the condi- 
tions in depth in the picture, the visualisation of the 
virtual instruments can be affected. Thus, the processing 
unit may know which parts of the virtual instruments have 

2 0 been covered by parts of the objects that are located in 
front of the instrument or the parts that are covered if 
the instrument has penetrated in an object. 

Advantageously, a form of lesson comprising a com- 
plete surgical procedure or operation can also be simu- 

2 5 lated. This is based on a number of procedures being pre- 

pared either for simulation or visualisation. When visu- 
alising, the video recordings are thus used which have 
been mentioned at the beginning of this section which 
shows a real filmed operation. 
30 It should be understood that a number of modifica- 

tions of the embodiment described above are possible 
within the scope of the invention; for example rendered 
portions of changes of the surface structure of the model 
can be placed in several layers on the video sequence. 

3 5 Such variants and similar variants must be considered 

comprised by the invention as defined by the appended 
claims . 



WO 02/059859 



PCT/SE02/00054 



13 
CLAIMS 

1. A method for simulating a surgical procedure on a 
5 bodily object, such as an organ, in a virtual environment 
provided by a computer unit, which environment comprises 

a three-dimensional mathematically described model 
of said bodily object, which at least reflects the geo- 
metrical and mechanical properties of said bodily object, 
10 and 

a virtual instrument which is controlled by a physi- 
cal feeding means connected to said processing unit, 
which makes it possible to affect said model, the method 
being characterised by the step of 

15 representing a two-dimensional projection of said 

model by means of a video sequence containing a recorded 
view of a real bodily object, 

engagement with the virtual instrument only inter- 
acting with said model. 

20 2. A method as claimed in claim 1, which comprises 

the further steps of 

providing a rendered surface of those and only those 
portions of said model which are affected by said virtual 
instrument , and 

25 superimposing said rendered surface on said video 

sequence . 

3 . A method as claimed in any one of the preceding 
claims, which comprises the further step of 

measuring the position of the light source relative 
3 0 to said bodily object when recording the video sequence, 
the step of rendering said rendered surface occurring by 
means of a light source having the same position. 

4 . A method as claimed in any one of the preceding 
claims, wherein said video sequence is recorded by a 
35 laparoscopic camera. 

5. A method as claimed in any one of the preceding 
claims, wherein said video sequence is a loop which cor- 
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responds to the natural movements of a real object during 
the initial state of the object. 

6. A method as claimed in any one of the preceding 
claims, which further comprises the steps of 

5 filming with a laparoscopic camera, from the same 

perspective, a real bodily object, and 

selecting a sequence from said recorded film that 
comprises a complete breathing and pulse cycle. 

7. A method as claimed in any one of the preceding 
10 claims, which comprises the further steps of 

determining which parts of the virtual instrument 
are completely or partly covered by the virtual object, 
and 

placing only the parts which are not covered by the 
15 virtual instrument on the top of the layer with said 
video sequence. 

8. A method as claimed in any one of the preceding 
claims, which comprises the further steps of 

measuring at least one parameter in the virtual in- 
20 strument's movements or interaction with the object, the 
parameter reflecting the user's skill. 

9. A method as claimed in claim 8, wherein the pa- 
rameter that is measur<ed is collision with tissues. 

10. A method for simulating a surgical operation 

25 containing a plurality of surgical procedures on a bodily 
object, such as an organ, in a virtual environment pro- 
vided by a computer unit, which environment comprises 

a three-dimensional mathematically described model 
of said bodily object, which reflects the geometrical and 

30 mechanical properties of said bodily object, and 

a virtual instrument which is controlled by a physi- 
cal feeding means connected to said processing unit, 
which makes it possible to affect said model, the method 
being characterised by the steps of 

35 representing a two-dimensional projection of said 

model by means of a video sequence containing a recorded 
view of a real bodily object, 
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providing said model in a plurality of states which 
reflect the object during a number of distinct phases 
during an operation, 

providing a first plurality of video sequences which 
5 correspond to the natural movements of a real object dur- 
ing the respective states, 

providing a second plurality of video sequences 
which each visualise the surgical procedure that shows 
how the object changes from one state to another, and 
10 allowing the user to choose between simulating a 

procedure and looking at a visualisation of the procedure 
with the purpose of changing the object from one state to 
another. 

11. A system for simulating a surgical procedure on 
15 a bodily object, such as an organ, in a virtual environ- 
ment provided by a computer unit, which environment com- 
prises 

a three-dimensional mathematically described model 
of said bodily object, which reflects at least the geo- 
20 metrical and mechanical properties of said bodily object, 
and 

a virtual instrument which is controlled by a physi- 
cal feeding means connected to said processing unit, 
which makes it possible to affect said model, the system 
2 5 being characterised by 

a two-dimensional projection of said model which is 
represented by a video sequence containing a recorded 
view of a real bodily object, 

engagement with the virtual instrument only inter- 
30 acting with said model. 
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